Cross validation for the classical model of structured expert judgment

We update the 2008 TU Delft structured expert judgment database with data from 33 professionally contracted Classical Model studies conducted between 2006 and March 2015 to evaluate its performance relative to other expert aggregation models. We briefly review alternative mathematical aggregation schemes, including harmonic weighting, before focusing on linear pooling of expert judgments with equal weights and performance-based weights. Performance weighting outperforms equal weighting in all but 1 of the 33 studies in-sample. True out-of-sample validation is rarely possible for Classical Model studies, and cross validation techniques that split calibration questions into a training and test set are used instead. Performance weighting incurs an “out-of-sample penalty” and its statistical accuracy out-of-sample is lower than that of equal weighting. However, as a function of training set size, the statistical accuracy of performance-based combinations reaches 75% of the equal weight value when the training set includes 80% of calibration variables. At this point the training set is sufficiently powerful to resolve differences in individual expert performance. The information of performance-based combinations is double that of equal weighting when the training set is at least 50% of the set of calibration variables. Previous out-of-sample validation work used a Total Out-of-Sample Validity Index based on all splits of the calibration questions into training and test subsets, which is expensive to compute and includes small training sets of dubious value. As an alternative, we propose an Out-of-Sample Validity Index based on averaging the product of statistical accuracy and information over all training sets sized at 80% of the calibration set. Performance weighting outperforms equal weighting on this Out-of-Sample Validity Index in 26 of the 33 post-2006 studies; the probability of 26 or more successes on 33 trials if there were no difference between performance weighting and equal weighting is 0.001

Quantifying uncertainty in intervention effectiveness with structured expert judgement : an application to obstetric fistula

To demonstrate a new application of structured expert judgement to assess the effectiveness of surgery to correct obstetric fistula in a low-income setting. Intervention effectiveness is a major input of evidence-informed priority setting in healthcare, but information on intervention effectiveness is generally lacking. This is particularly problematic in the context of poorly resourced healthcare settings where even efficacious interventions fail to translate into improvements in health. The few intervention effectiveness studies related to obstetric fistula treatment focus on the experience of single facilities and do not consider the impact of multiple factors that may affect health outcomes. We use the classical model of structured expert judgement, a method that has been used to quantify uncertainty in the areas of engineering and environmental risk assessment when data are unavailable. Under this method, experts quantify their uncertainty about rates of long-term disability in patients with fistula following treatment in different contexts, but the information content drawn from their responses is statistically conditioned on the accuracy and informativeness of their responses to a set of calibration questions. Through this method, we develop best estimates and uncertainty bounds for the rate of disability associated with each treatment scenario and setting. Eight experts in obstetric fistula repair in low and middle income countries. Estimates developed using performance weights were statistically superior to those involving a simple averaging of expert responses. The performance-weight decision maker's assessments are narrower for 9 of the 10 calibration questions and 21 of 23 variables of interest. We find that structured expert judgement is a viable approach to investigating the effectiveness of medical interventions where randomised controlled trials are not possible. Understanding the effectiveness of surgery performed at different types of facilities can guide programme planning to increase access to fistula treatment

Expert elicitation : using the classical model to validate experts' judgments

The inclusion of expert judgments along with other forms of data in science, engineering, and decision making is inevitable. Expert elicitation refers to formal procedures for obtaining and combining expert judgments. Expert elicitation is required when existing data and models cannot provide needed information. This makes validating expert judgements a challenge because they are used when other data do not exist, and thus measuring their accuracy is difficult. This article examines the Classical Model of structured expert judgment, which is an elicitation method that includes validation of the experts' assessments against empirical data. In the Classical Model, experts assess both the unknown target questions and a set of calibration questions, which are items from the experts’ field that have observed true values. The Classical Model scores experts on their performance in assessing the calibration questions and then produces performance-weighted combinations of the experts. From 2006 through March 2015, the Classical Model has been used in thirty-three unique applications. Less than one-third of the individual experts in these studies were statistically accurate, highlighting the need for validation. Overall, the performance-based combination of experts produced in the Classical Model is more statistically accurate and more informative than an equal weighting of experts

Analysis of the universal immunization programme and introduction of a rotavirus vaccine in India with IndiaSim

India has the highest under-five death toll globally, approximately 20% of which is attributed to vaccine-preventable diseases. India's Universal Immunization Programme (UIP) is working both to increase immunization coverage and to introduce new vaccines. Here, we analyze the disease and financial burden alleviated across India's population (by wealth quintile, rural or urban area, and state) through increasing vaccination rates and introducing a rotavirus vaccine. We use IndiaSim, a simulated agent-based model (ABM) of the Indian population (including socio-economic characteristics and immunization status) and the health system to model three interventions. In the first intervention, a rotavirus vaccine is introduced at the current DPT3 immunization coverage level in India. In the second intervention, coverage of three doses of rotavirus and DPT and one dose of the measles vaccine are increased to 90% randomly across the population. In the third, we evaluate an increase in immunization coverage to 90% through targeted increases in rural and urban regions (across all states) that are below that level at baseline. For each intervention, we evaluate the disease and financial burden alleviated, costs incurred, and the cost per disability-adjusted life-year (DALY) averted. Baseline immunization coverage is low and has a large variance across population segments and regions. Targeting specific regions can approximately equate the rural and urban immunization rates. Introducing a rotavirus vaccine at the current DPT3 level (intervention one) averts 34.7 (95% uncertainty range [UR], 31.7–37.7) deaths and $215,569 (95$207,846–$223,292) out-of-pocket (OOP) expenditure per 100,000 under-five children. Increasing all immunization rates to 90$45,914 (95% UR, $37,909–$53,920) OOP expenditure. Scaling up immunization by targeting regions with low coverage (intervention three) averts a slightly higher number of deaths and OOP expenditure. The reduced burden of rotavirus diarrhea is the primary driver of the estimated health and economic benefits in all intervention scenarios. All three interventions are cost saving. Improving immunization coverage and the introduction of a rotavirus vaccine significantly alleviates disease and financial burden in Indian households. Population subgroups or regions with low existing immunization coverage benefit the most from the intervention. Increasing coverage by targeting those subgroups alleviates the burden more than simply increasing coverage in the population at large

Reply to comment on "Suburban watershed nitrogen retention: Estimating the effectiveness of stormwater management structures" by Koch et al. (Elem Sci Anth 3:000063, July 2015)

We reply to a comment on our recent structured expert judgment analysis of stormwater nitrogen retention in suburban watersheds. Low relief, permeable soils, a dynamic stream channel, and subsurface flows characterize many lowland Coastal Plain watersheds. These features result in unique catchment hydrology, limit the precision of streamflow measurements, and challenge the assumptions for calculating runoff from rainfall and catchment area. We reiterate that the paucity of high-resolution nitrogen loading data for Chesapeake Bay watersheds warrants greater investment in long-term empirical studies of suburban watershed nutrient budgets for this region

Suburban watershed nitrogen retention : estimating the effectiveness of stormwater management structures

Excess nitrogen (N) is a primary driver of freshwater and coastal eutrophication globally, and urban stormwater is a rapidly growing source of N pollution. Stormwater best management practices (BMPs) are used widely to remove excess N from runoff in urban and suburban areas, and are expected to perform under a wide variety of environmental conditions. Yet the capacity of BMPs to retain excess N varies; and both the variation and the drivers thereof are largely unknown, hindering the ability of water resource managers to meet water quality targets in a cost-effective way. Here, we use structured expert judgment (SEJ), a performance-weighted method of expert elicitation, to quantify the uncertainty in BMP performance under a range of site-specific environmental conditions and to estimate the extent to which key environmental factors influence variation in BMP performance. We hypothesized that rain event frequency and magnitude, BMP type and size, and physiographic province would significantly influence the experts’ estimates of N retention by BMPs common to suburban Piedmont and Coastal Plain watersheds of the Chesapeake Bay region. Expert knowledge indicated wide uncertainty in BMP performance, with N removal efficiencies ranging from 40%. Experts believed that the amount of rain was the primary identifiable source of variability in BMP efficiency, which is relevant given climate projections of more frequent heavy rain events in the mid-Atlantic. To assess the extent to which those projected changes might alter N export from suburban BMPs and watersheds, we combined downscaled estimates of rainfall with distributions of N loads for different-sized rain events derived from our elicitation. The model predicted higher and more variable N loads under a projected future climate regime, suggesting that current BMP regulations for reducing nutrients may be inadequate in the future

Health and economic benefits of scaling up a home-based neonatal care package in rural India : a modelling analysis

Approximately 900 000 newborn children die every year in India, accounting for 28% of neonatal deaths globally. In 2011, India introduced a home-based newborn care (HBNC) package to be delivered by community health workers across rural areas. We estimate the disease and economic burden that could be averted by scaling up the HBNC in rural India using IndiaSim, an agent-based simulation model, to examine two interventions. In the first intervention, the existing community health worker network begins providing HBNC for rural households without access to home- or facility-based newborn care, as introduced by India’s recent programme. In the second intervention, we consider increased coverage of HBNC across India so that total coverage of neonatal care (HBNC or otherwise) in the rural areas of each state reaches at least 90%. We find that compared with a baseline of no coverage, providing the care package through the existing network of community health workers could avert 48 [95% uncertainty range (UR) 34–63] incident cases of severe neonatal morbidity and 5 (95% UR 4–7) related deaths, save $4411 (95$3088–$5735) in out-of-pocket treatment costs, and provide$285 (95% UR $200–$371) in value of insurance per 1000 live births in rural India. Increasing the coverage of HBNC to 90% will avert an additional 9 (95% UR 7–12) incident cases, 1 death (95% UR 0.72–1.33), and $613 (95$430–$797) in out-of-pocket expenditures, and provide$55 (95% UR $39–$72) in incremental value of insurance per 1000 live births. Intervention benefits are greater for lower socioeconomic groups and in the poorer states of Chhattisgarh, Uttarakhand, Bihar, Assam and Uttar Pradesh

Quantifying uncertainty about future antimicrobial resistance: Comparing structured expert judgment and statistical forecasting methods

The increase of multidrug resistance and resistance to last-line antibiotics is a major global public health threat. Although surveillance programs provide useful current and historical information on the scale of the problem, the future emergence and spread of antibiotic resistance is uncertain, and quantifying this uncertainty is crucial for guiding decisions about investment in antibiotics and resistance control strategies. Mathematical and statistical models capable of projecting future rates are challenged by the paucity of data and the complexity of the emergence and spread of resistance, but experts have relevant knowledge. We use the Classical Model of structured expert judgment to elicit projections with uncertainty bounds of resistance rates through 2026 for nine pathogen-antibiotic pairs in four European countries and empirically validate the assessments against data on a set of calibration questions. The performance-weighted combination of experts in France, Spain, and the United Kingdom projected that resistance for five pairs on the World Health Organization’s priority pathogens list (E. coli and K. pneumoniae resistant to third-generation cephalosporins and carbapenems and MRSA) would remain below 50% in 2026. In Italy, although upper bounds of 90% credible ranges exceed 50% resistance for some pairs, the medians suggest Italy will sustain or improve its current rates. We compare these expert projections to statistical forecasts based on historical data from the European Antimicrobial Resistance Surveillance Network (EARS-Net). Results from the statistical models differ from each other and from the judgmental forecasts in many cases. The judgmental forecasts include information from the experts about the impact of current and future shifts in infection control, antibiotic usage, and other factors that cannot be easily captured in statistical forecasts, demonstrating the potential of structured expert judgment as a tool for better understanding the uncertainty about future antibiotic resistance.Applied Probabilit